Skip to content

[Feat] Talk naturally with Fast sessions using GPT-Live - #1994

Merged
mrubens merged 20 commits into
developfrom
feat/live-voice
Sep 10, 2026
Merged

[Feat] Talk naturally with Fast sessions using GPT-Live#1994
mrubens merged 20 commits into
developfrom
feat/live-voice

Conversation

@mrubens

@mrubens mrubens commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

​Opened on behalf of @mrubens. Follow up by mentioning @roomote-roomote, in the web UI, or in Slack.

What changed

Voice is a phone call with Roomote, transcribed into the Session.

  • Adds a voice call to Fast Sessions from the Session composer, home page, and New Session flow. From the home page or dialog, the button opens a Session (sending anything already typed as the first message) and the call starts inside it.
  • Uses OpenAI gpt-live-1 for one native full-duplex WebRTC conversation with interruption handling. GPT-Live is Roomote's voice: it acknowledges each request in a few words, hands anything involving code or work to the Fast Session through client delegation, answers greetings and small talk itself, and reports results faithfully in its own words, keeping numbers, names, paths, and link labels exact.
  • Spoken requests run Fast turns in a new voice mode. The Fast prompt gains a Voice Call section (write for the ear, lead with the answer, exact values, no Markdown), and the reply is returned to GPT-Live as commentary, streamed sentence by sentence as it is written. Reply-stream chunks carry the Fast turn id, so each streamed sentence is attributed to the exact delegation that asked for it, even when a second request arrives first. In the transcript that reply renders as a collapsed Reported result to voice row rather than a chat bubble, so the exact result is still there if the call dropped before it was spoken. A call opened with text already typed runs that first turn in voice mode too, so its reply is spoken. Typed messages sent during a call are answered in writing as usual.
  • The call is transcribed into the Session as it happens: both sides appear word by word while they are spoken, then hand over to the persisted voice-turn rows (or to the reply for a delegated request). Voice turns are also added to Fast's conversation history so later requests can refer back to the call. Call started and Call ended · duration markers frame each call (new voice_call transcript event). The Session's selected model, tools, permissions, and transcript are preserved throughout.
  • Each spoken request is cleaned up by the deployment's helper model (filler, false starts, misheard terms) before it reaches Fast, using the deployment's repository, environment, and integration names as spelling vocabulary. GPT-Live is told those same names so it recognises them.
  • In-call controls in the composer: call timer, mute microphone, silence Roomote's audio, and end call. Short rising and falling tones mark the start and end of the call.
  • Creates Live sessions on the control plane through POST /v1/live/sessions; the OpenAI API key stays server-side and the browser receives only the negotiated SDP answer. Voice is opt-in through its own R_VOICE_OPENAI_API_KEY, with no fallback to the general OPENAI_API_KEY. Upstream error bodies are included in server errors, and GPT-Live errors, session close, and peer connection state are logged in the browser console.
  • Releases microphone, peer, data-channel, and audio resources immediately when the call ends, including during an in-flight connection setup. Errors surface as a toast.
  • Documents the call model and the R_VOICE_OPENAI_API_KEY requirement.

Why this change was made

GPT-Live provides the natural, interruption-friendly conversation layer while Roomote's existing Fast harness remains the authoritative task agent. Making the voice the author of spoken replies, and the transcript the record of the call, avoids the double answers that come from having Fast write a chat reply and GPT-Live speak at the same time, and matches how GPT-Live is designed to be used.

Impact

Users can talk to the same Fast Session they type to, hear its results, and read the call back later in the Session. Deployments need an OpenAI project with gpt-live-1 access and R_VOICE_OPENAI_API_KEY set; without it, the voice controls remain hidden.

Verification

Exercised end to end against real OpenAI Live from a local deployment: home-page button opens an empty Session and the call auto-starts there with the Call started marker; mute toggles; a typed message during the call is answered in writing and not spoken; ending the call writes the Call ended marker with its duration. Earlier in the branch: session handshake, spoken delegation into Fast, kickoff and reconnection flows.

Bugs found and fixed along the way: the tRPC schema trimmed the SDP offer's required trailing CRLF (OpenAI rejected every offer with failed to unmarshal SDP: EOF); the autostart flag constant lived in a client module and reached the server page as a client-reference stub; the autostart effect's guard did not survive React StrictMode.

Lint, types, knip, and the voice hook, composer, transcript, home, server, and command test suites pass.

@roomote-community

roomote-community Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

No code issues found. See task

  • Return only complete persisted Fast messages to GPT-Live, avoiding incomplete Markdown and duplicate streamed output.
  • Release microphone and peer resources held by an in-flight GPT-Live handshake immediately when the user ends voice.
  • Update the Fast session transcript test mock for voice.status and cover the voice integration.
  • Use only persisted transcript timestamps for the voice cutoff so optimistic client messages cannot suppress spoken replies under clock skew.
  • Stop the voice session when a pending structured-input request hides its controls.
  • Correlate each queued Fast result with its originating GPT-Live delegation so an overlapping or interrupted request cannot receive the wrong result.
  • Normalize each complete persisted Markdown message before splitting it into GPT-Live commentary chunks.
  • Preserve delegation correlation for streamed Fast reply sentences before their persisted turn ID arrives.
  • Return a typed prompt's initial Fast reply to the voice call opened from the home or New Session composer.
  • Keep a held GPT-Live acknowledgement from being persisted before its cleaned request is submitted to Fast.

Reviewed 83805bc

Comment thread apps/web/src/app/(sandbox)/sessions/[sessionId]/FastSessionTranscript.tsx Outdated
Comment thread apps/web/src/app/(sandbox)/sessions/[sessionId]/FastSessionTranscript.tsx Outdated
Comment thread apps/web/src/hooks/useLiveVoice.ts
Adds a hands-free voice mode to the session composer: microphone audio
streams to OpenAI realtime transcription over WebRTC (ephemeral,
transcription-scoped tokens minted server-side), completed utterances
send through the normal fastSessions.reply path, and replies are read
aloud via streaming gpt-4o-mini-tts PCM with barge-in on detected
speech. Voice rides R_VOICE_OPENAI_API_KEY with OPENAI_API_KEY
fallback; the key stays on the control plane and unset keys hide the
feature.
gpt-live-transcribe streams word-by-word deltas but rejects server-side
turn_detection, so the browser now runs a lightweight energy-based VAD:
a pause commits the audio buffer to finalize the utterance, and detected
speech interrupts reply playback (with a higher threshold while audio is
playing so speaker echo does not cut the agent off).
…ce in transcript tests

- Spoken replies now wait for the composite agent-working signal (send in
  flight, turn responding, response pending) to clear instead of the
  first visible assistant message, so a Fast progress kickoff no longer
  swallows the real result.
- useLiveVoice tracks a start generation: a stop() or repeat start()
  during the handshake makes the stale attempt release its mic and peer
  instead of activating after the user cancelled. The composer toggle
  also cancels a connecting handshake.
- FastSessionTranscript.client.test.tsx mocks voice.status and
  useLiveVoice, and adds coverage for the toggle and turn-settle speech.
Comment thread apps/web/src/app/(sandbox)/sessions/[sessionId]/FastSessionTranscript.tsx Outdated
Comment thread apps/web/src/app/(sandbox)/sessions/[sessionId]/FastSessionTranscript.tsx Outdated
- The spoken-reply cutoff is now the newest transcript timestamp at the
  moment voice starts instead of the browser clock, so a client clock
  running ahead of the server no longer silences every reply.
- A pending structured input request replaces the composer and its voice
  controls, so the conversation is stopped when one arrives rather than
  leaving the microphone open with no End control.
The voice loop waited for the whole Fast turn to finish, then sent the
entire reply to TTS as one request, so the user heard nothing until
closeout plus first-byte on a long synthesis.

- Speak incrementally: each completed sentence of a streaming reply is
  queued the moment it lands, the remainder when the persisted row
  finalizes it. A per-message cursor keeps the persisted row from
  repeating what the stream already said, and progress messages are
  read without ever skipping the result that follows.
- Pipeline TTS: speak() now appends to a queue drained by one loop that
  keeps two synthesis requests in flight ahead of playback, uses short
  (~400 char) requests for fast first byte, and coalesces sentences that
  arrive while a request is pending.
- Barge-in mutes the rest of the interrupted reply rather than pausing
  it (surfaced as an interruptions counter for the transcript).
- VAD pause 800ms -> 600ms; the server memoizes the OpenAI key lookup for
  30s since synthesis is now many small requests.
Comment thread apps/web/src/app/(sandbox)/sessions/[sessionId]/FastSessionTranscript.tsx Outdated
Comment thread apps/web/src/app/(sandbox)/sessions/[sessionId]/FastSessionTranscript.tsx Outdated
Comment thread apps/web/src/app/(sandbox)/sessions/[sessionId]/FastSessionTranscript.tsx Outdated
…omposers

The home page composer and the New Session dialog get the same live voice
toggle as the session composer. A session needs content to exist, so the
composer listens for the first utterance, starts the Fast session with it
(plus any typed text), and opens /sessions/<id>?voice=1. The session page
reads that flag, starts voice as soon as the deployment confirms it is
configured with no spoken cutoff so the first reply is read aloud, and
drops the flag from the URL so a reload does not restart the conversation.

- TaskPromptInput gains optional voice controls and a banner slot for the
  status strip; dictation is disabled while a conversation is active.
- useVoiceEnabled shares the status lookup between the transcript and the
  new-session form.
- Voice stays hidden for environment launches (those are tasks, not
  sessions) and on deployments without voice configured.
- Docs updated.
@roomote-roomote

roomote-roomote Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Updated the voice implementation to GPT-Live-1 and addressed the current review findings in c97783cc: ending an in-flight connection now releases media/peer resources immediately, and every delegated voice request carries a canonical Fast turn ID so overlapping results return to the correct Live delegation. The TextShimmer API mismatch was resolved in the preceding merge commit.

Focused client and server tests, type checks, lint, docs validation, and pre-push checks pass. The branch includes current develop, the PR description matches the shipped GPT-Live architecture, and the addressed review threads are resolved. No OpenAI voice credential was available in the task sandbox, so a real GPT-Live handshake, audible playback, and spoken interruption were not verified.

@roomote-roomote roomote-roomote Bot changed the title Live voice conversations for Fast sessions [Feat] Talk naturally with Fast sessions using GPT-Live Sep 10, 2026
- Stop trimming the WebRTC SDP offer in the tRPC input schema; the trailing
  CRLF is required and OpenAI rejected every offer with "unmarshal SDP: EOF".
- Move the voice autostart flag out of the client component module so the
  server page reads the string instead of a client-reference stub, and drop
  the StrictMode-hostile "already started" guard.
- Remove the Listening/End status strip and live caption; the voice button
  is the only control and errors surface as a toast.
- Clean each spoken utterance with a small non-reasoning model before it
  enters the transcript, falling back to the raw text on failure.
- Tell GPT-Live which repositories, environments, and integrations the Fast
  session can reach, and feed the same names to cleanup as vocabulary.
- Play short start and stop cues; the New Session handoff ends silently.
- Include OpenAI error bodies in thrown errors and log start failures.
Route transcript cleanup through the shared non-task inference path with
modelRole 'small' instead of calling OpenAI directly with a hardcoded
model. Cleanup now follows the operator's helper-model setting and works on
deployments whose inference is not on OpenAI; the OpenAI key is only needed
for the GPT-Live session itself.
…om the first utterance

- Resolve voice only from R_VOICE_OPENAI_API_KEY; drop the OPENAI_API_KEY
  fallback so enabling OpenAI for task inference never turns voice on and
  never bills a GPT-Live session unintentionally.
- Add a kickoff voice mode for the home page and New Session dialog: GPT-Live
  is told the first thing the person says is their request, delegates it at
  once, and stays silent. The new Session opens in voice mode and speaks the
  reply.
- Prime the new Session's conversation with the request that created it via a
  session-wide instructions append, since that conversation never heard it.
- Log GPT-Live errors, session-close, data-channel-close, and peer connection
  state transitions in the browser console.
…erance

- Instruct GPT-Live to read commentary word for word, in order, without
  paraphrasing or adding remarks, and to delegate every utterance including
  greetings and short remarks instead of answering on its own.
- Speak each reply sentence by sentence while it streams: completed
  sentences go out as soon as they exist and the persisted row finishes the
  tail, tracked per message so nothing is read twice.
- Add a silence-flush safety net in the voice hook: speech that GPT-Live
  never delegates is sent to Fast after 1.5s of quiet with a null delegation,
  and a delegation arriving right after is treated as already handled.
- Attribute streamed reply pieces to the most recent spoken request until
  the persisted row pins the exact turn.
Voice now follows the call model: GPT-Live is Roomote's voice, Fast is the
worker, and the Session transcript is the record of what was said.

- GPT-Live policy: acknowledge in a few words, delegate anything involving
  code or work, answer small talk itself, report results faithfully with
  numbers and names kept exact. No more verbatim reading or
  delegate-everything rules.
- Spoken requests run Fast turns in voice mode (new voiceMode flag through
  the reply command, durable replay, and answerFastAgentQuestion). The Fast
  prompt gains a Voice Call section, and the reply is marked voiceCommentary:
  it streams to GPT-Live sentence by sentence and renders as a collapsed
  "Reported result to voice" row rather than a chat bubble. Typed messages
  during a call are still answered in writing.
- The call is transcribed into the Session: what GPT-Live said (its output
  transcript) and what the person said when it answered directly are
  persisted as voice turns via voice.recordTurn, and added to Fast's history
  so later requests can refer to the call. Call started/ended markers are a
  new voice_call event type rendered as dividers with the duration.
- The home page and New Session dialog open an empty Session and start the
  call there, replacing the kickoff Live mode and context priming; a
  pre-typed message is sent as the first turn.
- In-call controls in the composer: timer, mute microphone, silence output.
Comment thread apps/web/src/trpc/commands/fast-sessions/index.ts
…ies exactly

- Show both sides of the call as they are spoken: GPT-Live's words and the
  person's words render as in-progress bubbles that fill word by word, stay
  while their voice-turn row is written, and hand over to the persisted row
  (or to the reply's optimistic row for a delegated request).
- Carry the Fast turn id on reply-stream chunks (`fastTurnId`) so streamed
  sentences are attributed to the delegation that asked for them, even when
  a second spoken request arrives before the first reply's persisted row.
  Removes the last-delegation fallback and the responding-window heuristic.
- Run a typed kickoff in voice mode when the Session is opened for a call,
  so the first reply is spoken; an empty voice Session schedules no turn.
GPT-Live acknowledges the instant it delegates, before the request has been
cleaned up and sent, so the acknowledgement row was persisted first and
sorted above the request. The hook now exposes how many utterances are
between speech end and delivery, and the transcript holds spoken turns while
a request is in flight, recording them once it has been sent.
GPT-Live's input transcript carries non-speech tags such as [chuckle] and
[tongue click]. Strip them from what the person said before it reaches the
live bubble, the recorded turn, or the Fast request; annotation-only speech
is not a turn.
Inferring the in-flight state from send status and queue length left a gap
between dequeue and the reply mutation. Count spoken requests from enqueue
until their mutation settles and flush held acknowledgements only when that
count and the cleanup count are both zero.
A [chuckle] in GPT-Live's output transcript can be cut in two when the
person starts talking; each half then missed the bracket matcher and was
recorded as its own spoken turn. Strip an unfinished tag at the end of a
turn and a tag tail at the start of the next.
@mrubens
mrubens marked this pull request as ready for review September 10, 2026 22:22
@mrubens
mrubens merged commit 2ec2a81 into develop Sep 10, 2026
18 checks passed
@mrubens
mrubens deleted the feat/live-voice branch September 10, 2026 22:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant